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Abstract 

Clinical  decision  support  (CDS)  systems  show  promise  for  enhancing  patient 
safety,  but  they  require  rigorous  evaluation  before  they  can  be  implemented 
widely.  We  developed  a  software  application  for  use  with  personal  digital 
assistants  (PDAs)  that  models  patient-specific  dose  responses  to  help  physicians 
predict  steady-state  warfarin  dosing  requirements  and  steer  patients  to  a 
therapeutic  level  of  anticoagulation  as  quickly  and  safely  as  possible.  We  also 
designed  a  randomized,  controlled  multi-site  trial  to  evaluate  the  effectiveness  of 
the  Warfarin  Dosing  and  Communication  System  (WARFDOCS)  in  reducing 
warfarin-related  errors.  Numerous  obstacles  delayed  implementation  of  the  CDS 
system  and  completion  of  the  trial.  To  better  understand  the  causes  that  led  to  the 
delay,  we  interviewed  key  informants  at  participating  hospitals;  reviewed  study 
protocols,  administrative  records,  and  meeting  minutes;  and  held  discussions  to 
review  the  data  and  their  interpretation.  Salient  themes  were  identified  by 
consensus  of  the  research  team  and  these  were  corroborated  by  key  informants. 
Four  major  themes  emerged.  First,  agreement  to  participate  in  the  trial  reflected 
very  different  levels  of  commitment.  Sites  participating  in  CDS  system 
evaluations  must  be  managed  actively.  Second,  the  enthusiasm  of  end-users  for  a 
CDS  system  was  derived  from  a  complex  calculus  of  perceived  benefits  and 
burdens.  Unfortunately,  the  most  relevant  appeal  (that  such  a  system  would 
markedly  improve  patient  safety)  could  not  be  made  in  advance  of  the  trial.  Third, 
research  changes  everything.  Valid  research  procedures  (e.g.,  informed  consent, 
randomization,  and  intrusive  data  collection)  may  be  necessary,  but  can 
themselves  affect  a  key  outcome  of  most  CDS  system  evaluations:  user  uptake. 
Fourth,  strong  “center  effects”  (i.e.,  the  CDS  system  proved  effective  at  some 
sites,  but  not  at  others)  should  be  expected.  If  “all  politics  is  local,”  then  much  of 
patient  safety  research  is  localized  as  well. 


Introduction 

Clinical  decision  support  (CDS)  systems  are  computer-based  tools  designed  to 
assist  clinicians  in  making  higher  quality  and  more  cost-effective  medical 
decisions.  They  have  been  defined  operationally  as  “active  knowledge  systems 
which  use  two  or  more  items  of  patient  data  to  generate  case-specific  advice.”1 
CDS  systems  are  proliferating  rapidly  and  exhibit  considerable  potential  for 
improving  the  realm  of  patient  safety.  However,  despite  the  consensus  opinion 
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that  rigorous  assessment  of  these  systems  is  imperative,  the  pace  of  evaluation  has 
lagged  far  behind  implementation.  One  Internet-based  product  catalog  alone  lists 
73  medical  expert  and  knowledge-based  systems,3  yet  published  articles  on 
clinical  computing  remain  “a  descriptive  feast  but  an  evaluative  famine.”4  This  is 
unfortunate,  given  the  high  capital  costs  of  CDS  systems  development  and  the 
real  possibility  that  such  systems  could  produce  harm  as  well  as  benefit.5 

Evaluators  of  CDS  systems  face  numerous  scientific  and  technical  challenges, 
including  the  application  of  suitable  controlled  designs  in  the  face  of  rapidly 
advancing  technologies,  difficulties  in  selecting  the  unit  of  allocation  and  analysis 
(i.e.,  patient  vs.  group  or  cluster),  meaningful  outcomes  measurement,  and  the 
interpretation  of  results.4  However,  researchers  face  equally  daunting,  albeit  less 
heralded,  pragmatic  challenges  in  their  efforts  to  implement  and  evaluate  CDS 
systems.  These  challenges  arise  from  the  organizational  context  and  the  human 
beings  working  in  it.6  As  Lorenzi  described  it,  “People  who  have  low 
psychological  ownership  in  a  system  and  who  vigorously  resist  its  implementation 
can  bring  a  ‘technically  best’  system  to  its  knees.”7  Understanding  the  roots  of 
resistance  is  critical  to  managing  both  uptake  (by  users)  and  assessment  (by 
evaluators). 

Warfarin  is  being  prescribed  to  more  than  two  million  Americans  and  is 
among  the  most  hazardous  of  drugs  commonly  prescribed  today. 8-13  Dosing  errors 
that  result  in  excessively  high  or  low  anticoagulant  effects  are  common, 
particularly  in  the  first  month  of  treatment.  Sources  of  correctable  dosing  errors 
have  been  identified  and  include  poor  dosing  judgment,  failure  to  recognize  well- 
described  drug  interactions,  and  failure  to  communicate  important  dosing 
information  to  the  patient  during  the  transition  between  hospital  and  outpatient 
followup.8’11'13’14 

Establishing  the  maintenance  (i.e.,  steady-state)  dose  of  warfarin  is  a  major 
clinical  problem.  The  between-patient  variation  in  daily  warfarin  dose 
requirements  is  large;  patients  can  require  anywhere  from  about  0.5  mg  per  day  to 
more  than  15  mg  per  day.  Given  today’s  frequently  short  hospital  stays, 
pharmacologically  sensitive  patients  may  be  discharged  from  the  hospital  with 
their  international  normalized  ratio  (INR)  still  rising.  By  the  time  they  are  seen  in 
the  outpatient  setting,  they  may  already  be  bleeding. 

Clinical  researchers  have  developed  several  approaches  to  enhancing  the 
safety  of  warfarin  therapy.  In  the  ambulatory  setting,  specialized  anticoagulation 
clinics  lead  to  more  consistent  anti-coagulation  and  fewer  adverse  outcomes; 
similar  benefits  have  been  achieved  through  an  Internet-based  “virtual  clinic.”15 
In  the  hospital,  proven  strategies  for  improving  warfarin  dosing  include  the  use  of 
dosing  nomograms,16’  17daily  consultations  with  a  pharmacist,18  and  various 
desktop  computer-based  decision  support  tools.19 

While  effective,  the  existing  approaches  to  anti-coagulation  control  are 
cumbersome  and  have  proved  difficult  to  implement  in  practice.  In  an  effort  to 
address  these  drawbacks,  members  of  our  research  group  created  the  Warfarin 
Dosing  and  Communication  System  (WARFDOCS).  At  the  core  of  WARFDOCS 
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is  a  computerized  decision-support  aide  for  the  initiation  of  warfarin  therapy20 
that  can  be  loaded  on  a  Palm  OS(I!-based  personal  digital  assistant  (PDA).  The 
PDA  platfonn  was  chosen  to  allow  users  greater  mobility  and  complete 
independence  from  existing  hospital  information  technology  infrastructures.  The 
WARFDOCS  system  requires  only  the  PDA  running  the  WARFDOCS 
application  and  an  infrared-enabled  printer  to  provide  its  entire  range  of  benefits. 
No  other  connections  (e.g.,  to  a  desktop  computer  or  network)  are  necessary. 

We  developed  a  framework  for  evaluating  the  WARFDOCS  program’s  effects 
on  patient  safety,  but  numerous  obstacles  slowed  and  sometimes  threatened  to  halt 
our  progress.  To  better  understand  the  causes  behind  the  delay,  we  interviewed 
key  informants,  examined  study  records,  and  held  team  meetings  to  assess  the 
data  and  various  interpretations  of  it.  Although  we  began  with  an  open-minded 
curiosity  with  regard  to  “what  went  wrong,”  our  inquiry  eventually  focused  on 
four  narrower  questions: 

1)  In  preparing  to  evaluate  a  CDS  system,  who  must  accept  and  endorse 
the  concept,  and  what  does  it  mean  when  they  do?  The  problem  in  the 
WARFDOCS  project  was  not  “getting  to  yes,”21  but  rather 
understanding  that  a  willingness  to  participate  could  reflect  different 
degrees  of  commitment,  relative  to  the  stakeholders’  perceptions  of 
benefits,  burdens,  and  roles. 

2)  What  factors  influence  enthusiasm  for  the  adoption  of  a  clinical 
decision  support  system?  Once  the  WARFDOCS  project  was 
underway,  we  observed  substantial  variations — within  and  between 
hospitals — in  the  manner  with  which  phannacists  embraced  the  new 
tool. 

3)  In  what  ways  does  the  evaluation  of  CDS  systems  conflict  with  their 
implementation ?  Implementing  electronic  decision-support  systems 
can  be  quite  taxing.  Users  must  become  familiar  with  new 
technologies  and  integrate  them  into  their  work  patterns.  Perfonning  a 
systematic  evaluation  of  a  new  CDS  system  imposes  additional 
burdens  that  may  undermine  initial  enthusiasm  for  what  is  perceived  as 
an  awkward  system,  but  may  in  fact  be  an  effective  and  efficient  new 
tool. 

4)  Given  the  importance  of  local  factors  in  clinical  decision  support 
system  implementations,  to  what  extent  are  the  results  of  a  CDS 
system  evaluation  generalizable  to  other  hospitals  and  settings?  The 
same  decision-support  paradigm  may  take  on  a  very  different  guise, 
depending  on  the  clinical  and  organizational  setting.  Even  tools  that 
seem  to  represent  a  known  quantity  that  is  context-independent  (e.g., 
our  PDA-based  application)  may  influence  care  in  a  variety  of  ways 
and  through  different  pathways  when  implemented  in  a  variety  of 
clinical  settings.  Is  a  CDS  system  implemented  in  two  different 
settings  still  the  same  CDS  system? 
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In  this  report,  we  will  describe  the  difficulties  encountered  in  the  launch  of  a 
multi-center  controlled  trial  of  WARFDOCS,  emphasizing  the  perceptions  of 
CDS  system  end-users  as  they  pertain  to  the  organizational  contexts  affecting  the 
implementation.  Our  focus  on  evaluation  is  motivated  by  the  recognition  that 
rigorous  assessment  of  health  information  technologies  is  essential  for  sifting  the 
worthwhile  from  the  useless,  and  ultimately  for  effectively  disseminating  high- 
value  interventions.  Our  findings  have  implications  for  evaluation  planning, 
interpretation,  and  generalizability. 


Background  and  methods 

The  WARFDOCS  clinical  decision  support  system 

The  WARFDOCS  program  is  a  PDA-based  system  that  provides  a  daily  dose- 
response  estimation  using  the  Bayesian  pharmacokinetic-pharmacodynamic 
model  previously  implemented  in  a  desktop  system  named  DrugCalc.®  22  The 
system  models  a  patient’s  response  to  warfarin  and  allows  clinicians  to  “test  out” 
various  dosing  plans  that  include  changes  in  the  current  day’s  dose  and  the 
subsequent  day’s  dose.  Pharmacists  can  use  the  system  to  recommend  a  course  of 
anticoagulation  care  that  minimizes  the  time  required  to  achieve  the  target  INR 
and  the  likelihood  that  the  INR  will  rise  to  a  value  above  3.5  (at  which  point  the 
risk  of  bleeding  increases  sharply).  A  Drug  Interaction  Knowledge  Base  on  the 
PDA  provides  information  on  clinically  meaningful  drug  interactions,  but  these 
interactions  are  not  incorporated  into  the  actual  computer  model. 

Our  randomized  controlled  trial  evaluating  the  effectiveness  of  WARFDOCS 
relied  heavily  upon  inpatient  clinical  pharmacists.  The  protocol  calls  for 
pharmacists  to:  1)  identify  potential  study  patients  using  the  pharmacy’s  computer 
system  (or  a  paper  system,  in  the  absence  of  a  computer);  2)  review  the  patient’s 
medical  record  and  then  complete  an  eligibility  form  stored  on  the  PDA;  3)  enroll 
consenting  patients  after  first  discussing  the  study  with  them  and  randomizing  the 
patients  to  either  the  WARFDOCS  group  or  the  standard  care  group;  4)  make 
daily  rounds  on  enrolled  patients,  entering  data  on  daily  warfarin  doses,  INR 
values,  and  concurrent  medications;  5)  use  the  PDA  to  make  a  daily  dosing 
recommendation  and  communicate  this  information  to  the  ordering  clinician;  and 
6)  print  out  an  Anticoagulation  Discharge  Summary  Report  and  deliver  it  to  the 
physician,  clinic,  or  hospital  assuming  responsibility  for  subsequent  outpatient 
anticoagulation  care,  on  the  day  of  the  patient’s  discharge.  This  process  was 
designed  to  enhance  the  safety  of  warfarin  use  by  ensuring  that  physicians 
maintained  dosing  control,  that  INRs  were  monitored  daily,  and  that  all  critical 
information  was  transferred  to  the  responsible  outpatient  provider. 

Evaluation  design 

In  the  ongoing  WARFDOCS  trial,  patients  at  each  participating  hospital  are 
assigned  randomly  to  either  the  WARFDOCS  system  or  to  usual  care.  The 
hospitals  were  chosen  for  convenience  and  were  not  matched,  so  randomization 
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by  site  was  not  possible.  The  principal  outcome  of  interest  is  the  proportion  of 
patients  who  experience  bleeding  (caused  by  over-anticoagulation)  or  thrombosis 
(caused  by  under-anticoagulation)  during  the  4-week  period  following  their 
enrollment  in  the  study.  Baseline  clinical  data  are  obtained  by  chart  review. 
Warfarin  doses  and  INRs  are  tracked  daily  throughout  the  hospital  stay  for 
intervention  and  control  patients;  however,  INR  recommendations  are  generated 
for  intervention  patients  only.  Outcomes  data  are  collected  via  chart  review  and 
follow-up  telephone  interviews  with  patients  are  conducted  30-60  days  after  their 
discharge  from  the  hospital. 

Settings 

Nine  hospitals  initially  agreed  to  participate  in  the  WARFDOCS  project.  The 
essential  characteristics  of  the  hospitals  are  summarized  in  Table  1.  Each  facility 
had  a  unique  history  and  culture  with  regard  to  warfarin  care.  Pharmacists  at 
Hospital  A  have  been  involved  in  warfarin  management  for  more  than  a  decade. 
Anti-coagulation  pharmacists  routinely  review  the  hospital  charts  of  the  vast 
majority  of  inpatients  started  on  warfarin,  and  are  available  for  additional 
consultation  as  needed.  Inpatient  pharmacists  at  Hospitals  B  and  C  consult 
regularly  with  physicians  on  clinical  issues;  however,  no  formal  warfarin  service 
exists  at  either  facility. 


Table  1.  Characteristics  of  hospitals  originally  committed  to  participate 


Hospital 

Type 

Location 

Number 
of  beds 

Final  status 

A 

University 

Urban  mid¬ 
sized  city 

540 

Currently  enrolling 
patients 

B 

Group  model  health 

maintenance 

organization 

Suburban 

350 

Currently  enrolling 
patients 

C 

Non-profit  community 
hospital 

Suburban 

254 

Currently  enrolling 
patients 

D 

Nonprofit  community 
hospital 

Semi-rural 

89 

Currently  enrolling 
patients 

E 

Non-profit  community 
hospital 

Urban  core  of 
a  rural  region 

267 

Currently  enrolling 
patients 

F 

Non-profit  community 
hospital 

Rural 

102 

Withdrew  before 
enrolling  patients 

G 

Non-profit  community 
hospital 

Urban  core  of 
a  rural  region 

195 

Withdrew  after 
change  of 
management 

H 

Non-profit  community 
hospital 

Urban  mid¬ 
sized  city 

208 

Withdrew  after 
reviewing  detailed 
scope  of  work 

1 

Non-profit  community 
hospital 

Urban  mid¬ 
sized  city 

304 

Withdrew  after 
reviewing  detailed 
scope  of  work 
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There  is  little  history  of  pharmacist-physician  interaction  at  Hospitals  D  and  E, 
with  regard  to  clinical  issues.  A  policy  at  Hospital  F  permitting  warfarin  orders 
“per  pharmacy”  (i.e.,  a  pharmacist  calculates  and  dispenses  the  appropriate  dose) 
once  existed,  but  was  abandoned  soon  after  its  implementation;  community 
physicians  are  currently  responsible  for  warfarin  dosing  and  all  other  aspects  of 
inpatient  prescribing.  Hospitals  G,  H,  and  I  dropped  out  of  the  study  before  any 
assessment  of  the  prevailing  work  relations  could  be  performed. 

Data  sources  and  analytic  approach 

This  paper  presents  the  results  of  a  qualitative  exploration  of  barriers 
encountered  during  the  implementation  phase  of  the  WARFDOCS  project.  Most 
of  the  data  for  this  analysis  were  collected  through  informal  discussions  with 
administrators,  pharmacists,  and  physicians  at  the  participating  hospitals  between 
the  spring  of  2002  and  the  fall  of  2003.  The  study  staff  maintained  detailed  logs 
and  field  notes,  which  were  reviewed  during  weekly  project  meetings.  In  addition, 
one  investigator  (RLK)  conducted  telephone  interviews  in  March  of  2004  with 
eight  pharmacists  from  five  of  the  participating  hospitals.  The  interviewer  used  a 
set  of  guiding  questions  and  probes  to  ensure  consistency  across  the  interviews. 
The  guiding  questions  were  open-ended  and  designed  to  elicit  the  pharmacists’ 
experiences  with  and  assessments  of  the  new  technology  and  its  impact  on  their 
work  as  well  as  the  effects  of  the  research  project  (and  its  related  tasks)  on 
practice.  The  interviews  averaged  30  minutes  (range:  20-40  minutes)  in  length 
and  were  audiotape  recorded  and  transcribed  for  analysis. 

Themes  were  identified  through  an  iterative  process  of  transcript  review.  A 
subset  of  team  members  each  reviewed  the  interview  transcripts  and  noted  the 
predominant  and  recurrent  themes  in  the  pharmacists’  definitions  of  and 
perspectives  on  the  ways  in  which  the  new  technology  changed  the  nature  of  work 
and  work  relationships.  This  included  the  technology’s  effect  on  working 
relationships  and  the  structure  and  flow  of  work.  The  project  team  members  then 
held  collective  discussions  on  emergent  themes  and  patterns  and  how  they  might 
be  categorized.  They  resolved  disagreements  regarding  categorization  through 
discussion,  and  by  noting  differences  in  opinion  among  team  members,  and 
through  reviews  of  positive  and  negative  examples  in  the  data.  Members 
developed  consensus  opinions  regarding  salient  categories  for  coding  based  on  the 
data  and  its  relevance  to  the  pharmacists’  use  of  PDAs  for  warfarin  dosing. 


Results  and  discussion 

The  nature  of  agreement 

(or  discovering  the  meaning  of  “yes”) 

During  the  development  of  the  grant  application  for  the  WARFDOCS  trial, 
the  principal  investigator  (RHW)  received  assistance  from  the  California  Institute 
for  Health  Systems  Performance  (CIHSP)  (www.cihsp.org)  in  identifying 
potential  evaluation  sites  across  the  Sacramento  area.  We  then  contacted  hospital 
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administrators,  who  in  turn  referred  us  to  phannacy  managers.  As  noted,  nine 
hospitals  initially  indicated  their  support  for  the  study  (Table  1). 

Making  the  sale:  a  matter  of  timing 

Nearly  a  year  transpired  between  the  submission  of  the  WARFDOCS  research 
application  and  the  receipt  of  funding.  Additional  time  elapsed  as  technical 
consultants  labored  to  transfer  the  Bayesian  predictor  program  from  a  PC  to  a 
PDA  platform.  Once  the  tool  was  ready,  project  staff  contacted  hospitals  to 
organize  meetings  with  top  management  and  the  chief  hospital  pharmacists.  The 
hospital  administrators  gave  these  meetings  a  relatively  low  priority,  so  they  took 
several  months  to  schedule. 

In  the  time  frame  between  negotiation  and  implementation,  some  of  the 
personnel  who  had  committed  to  the  study  either  left  their  respective  hospitals  or 
were  replaced.  A  consulting  firm  was  hired  by  one  of  the  participating  hospitals  to 
create  a  more  cost-effective  pharmacy  service.  The  resulting  reorganization  left 
WARFDOCS  with  little  internal  leadership  support,  and  the  hospital  withdrew 
from  the  study.  At  other  sites,  participants  had  to  be  re-acquainted  with  the  goals 
and  methods  of  the  project  before  the  study  could  be  initiated.  Simultaneously,  a 
regional  shortage  of  pharmacists  increased  the  work  demands  at  nearly  all  the 
sites,  making  pharmacists  less  inclined  to  cooperate. 

Biting-off  versus  chewing 

Orientation  meetings  enabled  the  study  staff  to  describe  the  aims  of  the  study 
to  the  selected  pharmacists  and  pharmacy  managers,  while  introducing  them  to 
the  PDA  technology.  While  not  everyone  who  might  be  affected  by  the  project 
was  part  of  these  discussions,  the  initial  participant  “buy-in”  was  broad  and 
seemingly  enthusiastic.  The  focus  of  the  early  meetings  was  the  best  clinical  use 
of  the  technology,  and  the  pharmacists  and  administrators  seemed  to  recognize  its 
potential.  But  as  the  discussions  shifted  to  the  details  of  implementation  and  those 
aspects  of  the  project  necessary  to  properly  evaluate  the  tool,  the  institutional 
commitments  began  to  falter. 

Hospitals  H  and  I  withdrew  their  support  once  the  logistical  demands  of  the 
project  became  evident.  Perceived  staffing  problems  at  these  hospitals — resulting 
from  the  regional  and  national  phannacist  shortage”  — were  so  severe  that 
phannacy  managers  decided  their  staffs  would  not  have  the  time  to  participate, 
despite  their  expressed  interest  in  warfarin  dosing  and  patient  safety.  Offers  of 
financial  compensation  (for  lost  time)  and  “in  kind”  support  (to  facilitate  the 
project  logistics)  were  to  no  avail. 

Phannacy  managers  at  two  other  previously  committed  hospitals  concluded 
that  the  WARFDOCS  trial  would  require  additional  personnel.  Project  funds  were 
made  available,  but  union  rules  and  other  administrative  baniers  precluded  the 
hiring  of  additional  hospital  staff.  Hospital  policies  also  prohibited  study  staff 
from  having  contact  with  patients  in  non-University  of  California-affiliated 
hospitals.  The  regional  crisis  involving  the  supply  of  phannacists  and  pressures  on 
administrators  to  implement  other  major  programs  substantially  degraded  the 
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priority  assigned  to  WARFDOCS.  Every  step  forward  thus  depended  upon 
research  project  staff.  Phannacy  administrators,  swamped  by  competing  demands, 
were  unavailable  to  help  solve  study-related  problems  from  the  inside.  The  project 
staff  grew  more  and  more  creative  in  their  plans  for  making  the  project  more 
attractive  to  hospital  sites. 

Details  of  the  protocol  implementation  and  specific  in-kind  support  for 
research  tasks  had  to  be  renegotiated  at  several  sites  to  re-secure  cooperation 
agreements.  It  was  not  that  the  clinical  sites  reneged  on  their  commitments. 

Rather,  details  presumed  to  be  trivial  or  irrelevant  were  left  unspecified  by  the 
parties  to  the  original  agreements,  and  administrators  and  staff  members  later 
charged  with  carrying  out  the  protocols  became  reluctant  or  expressed  an  inability 
to  provide  the  expected  level  of  service. 

Latent  stakeholders 

The  WARFDOCS  project  garnered  much  support  from  the  senior  leadership 
of  the  hospitals  we  approached,  but  attempts  to  secure  commitments  from  the 
more  shadowy  networks  of  “latent  stakeholders”  were  less  successful.  This  group 
included  pharmacy  managers  and  assistant  managers  faced  with  increasing  time 
and  staffing  pressures;  line  pharmacists  uncomfortable  with  the  PDA  technology, 
the  computerized  decision  support  system,  or  the  need  to  interact  more  directly 
with  physicians;  and,  in  the  case  of  one  hospital,  physicians  with  no  historical 
precedent  for  collaboration  with  the  pharmacy  service. 

Barriers  to  adoptability:  pharmacists’  perceptions 

Although  this  paper  focuses  on  factors  affecting  the  implementation  and 
generalizability  of  the  WARFDOCS  evaluation,  it  is  impossible  to  ignore  the 
reaction  of  the  trial  participants  to  the  WARFDOCS  CDS  system  itself.  The 
ultimate  safety  benefit  delivered  by  a  CDS  system  depends  upon  not  only  its 
capacity  to  shape  safer  care,  but  also  the  extent  to  which  the  system  is  used.  CDS 
systems  disuse  has  been  a  longstanding  concern.-  ’  "  We  found  that  users’ 
perceptions  of  the  WARFDOCS  CDS  system’s  value  revolved  around  benefits, 
burdens,  and  time  demands.  Surprisingly,  the  changing  role  relationships  of 
phannacists  and  physicians  were  not  a  major  issue  for  the  users. 

Benefits  and  burdens 

Enthusiasm  for  the  WARFDOCS  CDS  system  itself  (as  opposed  to 
enthusiasm  for  the  project)  appeared  to  reflect  a  careful  calculus  of  perceived 
benefits  and  burdens,  which  varied  by  pharmacist  experience  and  the  prevailing 
social  norms  governing  interactions  between  phannacists  and  physicians.  The 
potential  benefits  of  a  computerized  warfarin  dosing  assistance  program  were 
most  clear  to  clinical  pharmacists  who  had  relatively  little  experience  with 
warfarin  and  who  wanted  to  assist  the  physicians  with  warfarin  dosing 
responsibilities.  They  regarded  the  program  as  an  enhancement,  enabling  them  to 
deliver  better,  safer  care,  and  perhaps  to  interact  more  effectively  with  the 
physicians. 


402 


PDA-based  Warfarin  Dose  Calculator 


First,  in  general...  I  don  7  think  we  were  doing  it  (warfarin) 
consistently  or  systematically ...  at  all.  And  I  think...  that’s 
probably  what  attracted  me  to  this.  I  think  that  patients  on 
warfarin — whether  it  was  cardiac,  ortho,  or  whatever — I  think  we 
were  just  being  taken  care  of  by  the  physicians,  and  if  the  patient 
looked  like  they  were  getting  in  trouble  with  their  INRs,  then  the 
pharmacists  would  intervene  and  make  a  call  and  say,  you  know, 

7  am  not  sure  what 's  going  on  here  but  how  can  I  help.  ’ 

— pharmacist  at  Hospital  C 

More  experienced  pharmacists  anticipated  fewer  gains,  and  a  few  expressed 
doubts  with  regard  to  the  trustworthiness  of  the  computer-driven 
recommendations: 

I  think  the  program  is  helpful  when  you  have  a...  fair  number  of 
data  points  in  it.  But  initially...  I  trust  my  own  instincts  because  I 
know  there  is  a  lot  of  stuff  that  does  not  go  into  the  PDA  that  is  in 
the  chart  and  is  in  my  head.  (The  software)  doesn ’t  know  about 
changes  in  antibiotics ...  advanced  age,  and  albumin,  and  I  know 
that.  There  are  a  few  things  that  are  in  there,  but  there  are  a  lot  of 
other  things  that  don  ’t  go  into  the  program.  — pharmacist  at 
Hospital  A 

The  perceived  burdens  of  using  the  program  varied  according  to  the 
phannacist’s  prior  experience  with  PDAs,  the  frequency  of  current  interactions 
with  physicians  and  their  offices  involving  warfarin,  and  practical  workplace 
details  such  as  the  physical  size  of  the  hospital,  the  location  of  patients  taking 
warfarin,  and  the  phannacist  staffing  and  schedules. 

I  have  been  involved  with  the  study  pretty  much  since  day  one,  and 
it’s  getting  easier  to  use  it  now...  Fm  a  little  more  adept  at  using 
it...  but  it  still  isn ’t  something  I  would  normally  pick  up  if  I  didn ’t 
have  to.  — pharmacist  at  Hospital  A 

Because  Fm  a  rotating  pharmacist,  I  don  7  have  a  certain  shift. 

(T)here  are  about  five  pharmacists  who  are  doing  this  (protocol), 
and  each  of  us  will  probably  do  it  for  three  times  a  week  or  maybe 
none  at  all  for  one  week.  It  has  to  do  with  our  scheduling  and 
staffing  problems.  — pharmacist  at  Hospital  B 

Pressures  of  time:  how  fast  is  fast  enough? 

WARFDOCS  is  a  tool  designed  to  improve  the  quality  of  clinical  decisions 
related  to  warfarin  dosing.  The  time  it  took  to  use  the  tool  was  a  source  of 
occasional  consternation.  Commenting  on  the  time  required  to  input  data  and  the 
delay  that  occurs  when  WARDOCS  is  generating  predictions,  one  experienced 
anti-coagulation  pharmacist  complained  early  on: 

Well,  to  be  brutally  honest...  it’s  become  murderously  slow  to  do 
my  job.  Sometimes  when  I’d  have  four  or  five  intervention  patients 
to  process  through  the  Palm  Pilot,  it  would  take  close  to  an  hour 
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just  to  put  in  their  data  points  and  come  up  with  predictions ...  If 
there  were  an  open  window  on  the  6th  Floor,  that  Palm  Pilot  would 
be  out  on  the  street  right  now.  — pharmacist  at  Hospital  A 

Efforts  to  expedite  the  data  processing  (through  the  use  of  a  more  powerful 
PDA)  mitigated  the  problem  substantially,  but  did  not  abolish  it.  Using  the 
WARFDOCS  system  simply  takes  longer  than  making  an  off-the-cuff  clinical 
decision.  Since  the  potential  benefits  of  WARFDOCS  are  spread  across  the  entire 
population  of  warfarin  patients,  it  is  not  surprising  that  pharmacists  had  a 
tendency  to  accentuate  the  (known)  burdens  while  discounting  the  (potential) 
benefits. 

interactions  with  physicians 

Hospitals  that  participated  in  the  study  varied  with  respect  to  the  norms  and 
standards  governing  interactions  between  pharmacists  and  physicians  involving 
warfarin.  In  general,  physicians  appeared  to  welcome  increased  pharmacy 
involvement. 

I  have  to  say  that  most  of  the  time  they  (the  physicians)  are  more 
than  happy  not  to  dose  Coumadin ...  more  than  happy  to  have  us  do 
it  rather  than  do  it  themselves.  — pharmacist  at  Hospital  B 

The  impact  of  WARFDOCS  on  pharmacist-physician  relations  varied  widely. 
For  example,  pharmacists  at  Hospital  A  have  been  involved  in  warfarin  dosing 
since  1992.  The  addition  of  a  handheld  decision  aide  at  this  hospital  had  little 
impact  on  established  patterns  of  interaction. 

I  don ’t  think  it  changed  interaction  with  physicians  at  all.  You 
know,  if  we  don ’t  think  the  dose  they  are  ordering  is  appropriate, 
we  ’ll  call  them  and  change  it,  or  we  ’ll  be  even  more  proactive  and 
recommend  a  dose...  prior  to  them  even  writing  it...  We  do  that 
everyday  anyway.  — pharmacist  at  Hospital  A 

At  other  hospitals,  however,  the  expectation  that  pharmacists  would 
communicate  regularly  with  ordering  physicians  represented  a  pronounced 
departure  from  established  patterns.  Although  challenging,  such  interactions  could 
have  an  upside: 

Before,  we  weren  'l  even  talking  about  dosing  at  all  with  the 
doctors...  Before,  all  we  would  do  is  enter  the  warfarin  order...  I 
personally  would  rather  recommend  a  dose  than  enter  the  order... 

It  is  a  little  bit  more  interesting.  — pharmacist  at  Hospital  B 

In  this  sense,  WARFDOCS  not  only  demanded  that  pharmacists  leam  new 
technical  skills,  it  also  expanded  their  professional  role  in  new,  and  sometimes 
rewarding,  ways. 

The  added  burdens  of  research 

The  evaluation  of  clinical  informatics  tools  is  essential  to  the  field’s  progress, 
yet  evaluation  tasks  make  demands  on  participants  that  are  not  integral  to  the 
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CDS  system  itself.  If  and  when  WARFDOCS  is  integrated  into  clinical  practice, 
future  clinician-users  will  be  expected  to  review  patients’  medical  records  to 
identify  current  medications,  medical  conditions,  warfarin  doses,  and  INR  values. 
They  also  will  be  required  to  generate  a  dose  prediction  and  make  a  subsequent 
decision  or  recommendation.  Outside  of  the  research  setting,  however,  future 
users  will  not  need  to  determine  eligibility  for  the  study,  obtain  consent  from 
patients  and/or  families,  or  enter  detailed  information  on  demographics  and 
medications.  Furthermore,  they  will  not  interact  regularly  with  the  50  percent  of 
patients  in  the  control  ann  of  the  study.  In  agreeing  to  participate  in  the  study,  the 
phannacists  could  not  have  meaningfully  anticipated  the  roughly  50  percent  of  the 
total  workload  that  would  be  devoted  to  research  overhead,  rather  than  clinical 
care.  Obtaining  infonned  consent  from  patients  was  viewed  as  particularly  taxing. 

Well,  the  other  main  stumbling  block  was  the  process  for  consen  t. 

It  was  something  that  pharmacists  hadn  7  clone  in  the  past...  and 
having  them  take  on  extra  things  they  ’re  not  used  to  doing,  was 
kind  of  a  hard  sell.  — pharmacist  at  Hospital  C 

These  observations  form  the  outline  of  a  paradox.  CDS  systems  need  to  be 
evaluated.  An  important  part  of  any  meaningful  evaluation  is  whether  the  system 
will  be  used.  And  yet  the  more  carefully  the  system  is  evaluated,  the  greater  the 
burden  of  initial  adoption,  and  the  less  likely  potential  users  are  to  embrace  the 
technology  itself.  Moreover,  attempts  to  solve  the  problem  by  embedding  research 
personnel  on  site  to  handle  logistical  issues  (as  we  did  at  select  hospitals)  can  in 
itself  be  problematic,  as  the  presence  of  outsiders  may  alter  the  way  in  which  the 
system  is  used. 

Limits  to  generalizability 

At  the  time  of  this  writing,  we  do  not  know  if  WARFDOCS  is  having  a 
beneficial  effect  on  warfarin-related  care  and  patient  outcomes  at  the  participating 
hospitals.  That  conclusion  awaits  additional  data  collection  and  analysis.  But  let 
us  assume  that  WARFDOCS  is  making  care  safer,  on  average,  within  the 
participating  hospitals.  In  a  moment  of  unrestrained  optimism,  let  us  even  assume 
that  WARFDOCS  is  associated  with  a  decrease  in  the  rate  of  major  hemorrhage, 
and  that  the  associated  R-value  is  less  than  0.05.  Finally,  let  us  assume — based  on 
the  randomized  design,  completeness  of  follow-up,  and  the  blinded  analysis — that 
the  results  can  be  trusted  (i.e.,  that  internal  validity  is  assured).  But  what  about 
external  validity?  To  what  extent  can  the  results  be  generalized  to  other  patient 
care  settings? 

We  approached  this  question  by  reviewing  how  WARFDOCS  was 
implemented  at  the  five  hospitals  that  actually  enrolled  patients  (Table  2).  We 
found  notable  variations  between  hospitals  along  three  dimensions:  1)  the 
phannacists’  prior  involvement  with  warfarin  care,  2)  the  degree  of  clinical 
interaction  between  pharmacists  and  physicians,  and  3)  workflow  and  logistical 
issues  affecting  implementation. 
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Table  2.  Between-hospital  variations  affecting  generalizability 


Hospital 

Prior  inpatient 
pharmacist 
involvement 
with  warfarin- 
related  care 

Prior  pharmacist- 
physician  clinical 
interactions 

Workflow  and  logistical 
factors  affecting 
implementation 

A 

Extensive 

Well-established 

Disruption  minimized  by 
assigning  WARFDOCS 
duties  to  the  “warfarin 
pharmacist”  of  the  day 

B 

Minimal 

Minimal  in  the  inpatient 
setting  (a  pharmacy-run 
warfarin  clinic  handles 
about  50%  of  outpatients) 

Pharmacists  “assigned  to 
station”  and  cannot  leave 
without  coverage 

C 

Moderate; 
pharmacists 
review  INRs  and 
potential  drug 
interactions 

Minimal 

Pharmacists  work  3-4 
shifts  per  week, 
necessitating  frequent 
handoffs  of  WARFDOCS 
responsibilities 

D 

Moderate;  hospital 
computer  system 
allows  pharmacist 
to  follow  warfarin 
doses  and 
subsequent  INRs 

Minimal 

Substantial  physical 
distance  between 
pharmacy  computer 
(source  of  dosing 
information)  and  nursing 
station  (source  of 
laboratory  data) 

E 

Moderate; 
extensive  in  six 
months  prior  to 
launch  of 
WARFDOCS 

Moderate;  warfarin 
declared  a  “quality 
improvement  focus”  by 
parent  hospital  chain  in 

2003,  leading  to  “warfarin 
per  pharmacy”  in  about 
half  of  patients 

PDAs  already  in  wide  use 
(e.g.,  for  assessing  drug 
interactions),  but  this 
created  own  problems; 
pharmacists  complained 
about  multiple  PDAs 

Informatics  tools  are  subject  to  substantial  center  or  implementation  effects, 
and  these  effects  are  not  easy  to  explain  or  account  for.  They  affect  the  quality  of 
care  by  changing  the  processing  and  flow  of  information — in  short,  by  changing 
the  content  and  patterns  of  communication.  Looked  at  in  this  way,  the 
WARFDOCS  intervention  is  not  a  single  clinical  trial  but  rather  five  related 
trials — one  for  each  hospital.  In  hospitals  like  A  and  E,  WARFDOCS  is 
principally  a  computerized  dosing  guide  that  imposes  additional  rigor  on  an 
existing  communication  process  between  pharmacists  and  physicians.  In  the  other 
hospitals,  WARFDOCS  provides  new  infonnation  and  encourages  a  whole  new 
mode  of  interaction  between  physicians  and  pharmacists.  There  is,  therefore,  no 
reason  to  expect  that  the  effect  of  WARFDOCS  will  be  homogeneous  across  all 
participating  centers.  Heterogeneity  of  effects  is  not  unusual  even  for  biomedical 
treatments.  In  one  notable  example,  the  Beta-Blocker  Heart  Attack  Trial  included 
21  dominant  centers  (i.e.,  the  effect  of  beta-blockers  was  the  same  as  for  the 
overall  study)  and  10  divergent  centers  (i.e.,  the  effect  of  beta  blockers  was 
deleterious).'  Recognition  of  so-called  center  effects  creates  an  imperative  for 
reporting  not  only  the  estimated  average  effect  of  the  treatment,  but  the  estimated 
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treatment  effect  dispersion  as  well.27  One  approach  is  to  fit  hierarchical  regression 
models  that  incorporate  centers  (sites)  as  random  effects.  Studies  in  which  large 

“site  effects”  are  anticipated  should  be  appropriately  powered  to  test  for  site- 

28 

treatment  interactions  and  for  patient  clustering  within  the  sites. 

We  suspect  that  WARFDOCS  is  not  the  sole  clinical  decision  support  system 
for  which  these  observations  are  true.  What  may  appear  to  be  “random  error”  in 
the  analysis  of  evaluation  data  may,  in  fact,  be  logically  related  to  variations  in  the 
organizational  context  that  are  influenced  or  made  manifest  as  a  direct  result  of 
the  CDS  system  implementation.  The  implementation  of  even  relatively  simple 
CDS  systems,  like  WARFDOCS,  requires  the  use  of  creative  efforts  to  overcome 
barriers  unique  to  each  site.  These  malleable  contextual  dimensions,  including 
clinical,  social,  and  organizational  factors,  may  be  what  makes  or  breaks  a  project 
at  any  given  site.  In  light  of  this,  it  may  be  that  the  “quality  of  implementation”  or 
even  the  “goodness  of  fit”  between  the  site  and  the  tool  rise  to  the  level  of 
variables  to  be  analyzed  rather  than  nuisances  to  be  randomized  away  in  the 
search  for  an  overall  “generalizable”  finding. 


Conclusions 

Our  experience  to  date  with  the  WARFDOCS  project  suggests  four  principal 
conclusions. 

•  Achieving  participatory  “buy-in”  for  a  CDS  system  evaluation  is  an 
active  process  that  requires  investments  of  time,  energy,  and  resources 
well  beyond  those  typically  budgeted  for  such  purposes.  Unless  there 
is  a  miraculous  alignment  of  organizational  priorities  and  study 
objectives  (such  as  that  we  were  fortunate  enough  to  experience  with 
Hospital  E),  investigators  and  funders  should  be  realistic  about  the 
need  to  actively  manage  organizational  relationships  and  budget  the 
work  accordingly. 

•  From  an  end  user’s  perspective,  the  acceptability  of  a  CDS  system 
reflects  a  calculus  of  benefits  and  burdens.  Resistance  to  the  system 
can  be  overcome  by  making  the  users’  jobs  more  interesting  or 
satisfying,  minimizing  disruptions  in  workflow,  and  perhaps  most 
importantly,  appealing  to  their  professional  commitment  to  patient 
safety.  The  most  relevant  appeal  would  involve  a  clear  demonstration 
of  the  patient  benefit,  which  cannot  be  accomplished  until  the  end  of 
the  trial. 

•  A  CDS  system  implementation  may  be  greatly  complicated  by  a 
formal  system  evaluation,  especially  if  the  evaluation  is  a  randomized 
controlled  trial.  In  the  context  of  such  a  trial,  there  is  no  obvious  way 
around  the  additional  burdens  imposed  by  the  need  to  obtain  consent, 
collect  data,  and  involve  control  patients  (who  must  complete  all  the 
study  procedures,  but  will  reap  no  benefit).  These  procedures  are 
necessary  but  may  themselves  influence  a  key  CDS  system  evaluation 
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outcome:  user  uptake.  There  is  no  ready  solution  other  than  to  describe 
the  research  procedures — and  their  perceived  impact  on  end  users — as 
completely  as  possible,  so  the  results  of  the  evaluation  may  be 
assessed  in  the  proper  context. 

•  In  the  evaluation  of  CDS  systems,  strong  “center  effects,”  manifested 
as  qualitative  interactions  (i.e.,  positive  effects  at  some  sites  and 
negative  effects  at  others)  should  be  expected.  This  is  because 
informatics  technologies  always  are  embedded  in  a  clinical  and  social 
context  that  can  vary  substantially  from  site  to  site.  Aside  from  raising 
philosophical  questions  about  the  results  of  CDS  systems  trials  and 
whether  they  are  ever  “generalizable,”  this  observation  underscores  the 
importance  of  using  a  mixture  of  methods  (i.e.,  quantitative  and 
qualitative)  to  characterize  potentially  important  contextual  factors  as 
thoroughly  as  possible. 

In  summary,  when  implementing  tools  to  improve  patient  care,  the  experience 
of  the  journey  may  be  as  important  as  the  ultimate  destination. 
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